ADFA-5241: Declare utf-8 on text responses from the documentation database - #1725

…abase

WebServer wrote the stored MIME type into Content-Type verbatim, and no
ContentTypes.value in documentation.db carries a charset, so every text
response left its encoding unstated. A client that does not assume UTF-8 falls
back to a legacy single-byte encoding: of 500 sampled text rows, 328 (66%)
contain non-ASCII bytes with no BOM, and those render as mojibake. Observed on
device while confirming ADFA-5239 -- a/android/R.id.html shows
"   ↳android.R.id" where the page says "↳android.R.id".

Two things made this invisible until now. Those pages were being served as
text/plain (ADFA-5239), so they rendered as source rather than as documents,
and Android's WebSettings.defaultTextEncodingName defaults to UTF-8, which
likely spares the in-app viewer -- nothing in the app sets that property, so it
relies on the default. Neither is a reason to leave the header unstated:
correctness by client default is per-client behaviour, not something this
server declares, and the same file already sends "; charset=utf-8" on its
hardcoded responses (lines 1176, 1225, 1247, 1310). Only database-sourced
content was missing it.

The predicate lives in common/ContentTypeHeaders rather than in WebServer,
because ADFA-5176's in-process transport already answers this question on its
own -- DocumentationRequestInterceptor.mimeAndCharset defaults text/* to utf-8
-- and the two transports disagreeing about what a response says is worse than
either answer. That branch should adopt this on its rebase.

What gets a charset, and why not more: every text subtype, including the
database's malformed bare "text" (which 726 TooltipButtons reach via x.html)
and "text/text"; XML-based types, since SVG usually omits its own declaration
and a transport charset takes precedence anyway. Not application/json -- RFC
8259 defines no charset parameter for it and fixes the encoding as UTF-8, so
declaring one says nothing; there is a test asserting that, so nobody "fixes"
it later. An already-declared charset is never doubled.

Deliberately not fixed in the database. Putting the parameter in
ContentTypes.value would work for both readers, but that column doubles as a
lookup key matched exactly by ExtensionToContentTypeResolver (the plugin
installer would skip every HTML asset), by docdb-studio's anchor extraction,
and by three OfflineDocumentationTools scripts. Two of those fail silently.
The analysis is on ADFA-5241.

Tests: 7 for the helper -- text, the malformed types, XML, the binary set, the
JSON exclusion, no doubling, and casing/parameter tolerance -- plus one that
asserts the header a real client receives, since the helper being right does
not prove the response is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…undaries

Both from CodeRabbit on PR #1725.

`type.startsWith("text")` classified `textual/example` as a text response. The
comment beside it already said the intent was "every text subtype, plus the
bare text oddity" -- the code just did not say that. Now `type == "text" ||
type.startsWith("text/")`.

`mimeType.contains("charset=")` found the substring inside *another*
parameter's value, so `text/html; note="charset=utf-8"` looked like it already
declared an encoding and got none added. Parameters are now split on `;` and
matched by name, which also keeps `text/html;boundary=x` working.

Neither case exists in documentation.db today -- no ContentTypes.value carries
a parameter at all -- so this is about the helper being honest rather than a
live defect. Both have regression tests.

Also dropped the non-ASCII from the KDoc, which quoted the mojibake it was
describing. The ASCII policy exempts a glyph doing real visual work, and I had
read the example as qualifying; naming the code point instead reads the same,
which means it does not qualify. The file is now pure ASCII.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nts as text

From the code review on PR #1725. Four correctness findings, all real.

Splitting on ';' still finds "charset=" inside a quoted value that contains a
semicolon -- text/html; note="x; charset=utf-8" -- so the helper concluded a
charset was already declared and sent none at all: the exact failure its own
KDoc claimed the split prevented. And a parameter with no value (; charset) or
an empty one (; charset=) was read as a declaration, with the same result.
Parameters are now parsed with quote awareness, and only a charset parameter
with a non-empty value counts as declared.

The textual application/* list omitted application/javascript, which does have
rows and is what ExtensionToContentTypeResolver maps ".mjs" to, so real files
served undeclared -- the bug this class exists to prevent. Added it along with
ecmascript and x-sh, and said in the comment that the list is a list precisely
because these types share no marker, so anything textual arriving later has to
be added rather than assumed covered.

The header value is now built before the status line goes out. The writer
autoflushes, so a throw after the first println made sendError append a second
status line to a response that already claimed 200, which a client parses as a
malformed header rather than as an error. dbMimeType is a platform type from
Cursor.getString, so a NULL ContentTypes.value is a real way to reach that
throw.

typeAndCharset is exposed because ADFA-5176's interceptor needs the type and
the charset apart for WebResourceResponse and was re-implementing the parse to
get them -- with the naive substring match this file warns against. The class
was created so both transports answer alike; keeping the parse private meant
they agreed on the default and disagreed on reading what was already there.

The "two thirds" statistic is replaced with a full census of the database it
was measured on: 17,903 of 29,139 text rows, 61.4%. The review measured 22.5%
against the bundled asset, which is an older export -- both numbers are right
for their own database, so the KDoc now names which one and says the rate is
per-generation.

Four regression tests for the parsing cases, one for the textual types, one for
typeAndCharset. The header assertion now prints the whole response when no
Content-Type is found instead of throwing NoSuchElementException, and no
longer declares a compression-dictionary version it does not use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…med inputs

Both from the re-review on #1725.

An escaped quote no longer ends a quoted parameter value. RFC 9110's quoted-pair
means \" does not close the string, but the parser toggled on every quote, so
text/html; note="a\"; charset=iso-8859-1 parsed as two parameters and the
charset inside note read as a declaration -- the same false match this class
exists to prevent.

An empty charset parameter is now left alone rather than contradicted.
headerValue used to append a second charset, producing
text/html; charset=; charset=utf-8. Recipients keep an empty valued parameter
and ignore a repeated name, so that append claims a fix it does not make. A
parameter with no = at all is still appended to, because that one really is
dropped during parsing, so the appended charset takes effect. typeAndCharset
keeps substituting the default either way -- it hands the charset back as its
own value, where nothing can conflict with it.

Both tests were confirmed to fail against the previous parser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Its KDoc said null meant the type already carried a charset, without saying that
an empty one counts -- which is the interesting case, and the reason
typeAndCharset answers differently for the same input. Both now state the
asymmetry and why it exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ored

Review of this PR found that appending a charset to the stored string leaves the
header no better formed than the database happened to be -- and that on the row
this change was written for, it does nothing at all.

The database stores a bare "text" (one row, x.html) and a "text/text" (ten
licence files). Neither is a media type -- type "/" subtype is required -- and a
client that cannot parse the type discards the charset with it, so those rows
went on being sniffed exactly as before. Both normalize to text/plain now.

A stored value carrying a control character was written into the response
verbatim by println(). ContentTypes.value comes from a database that a debug
build swaps in from shared storage, so a planted value containing CR/LF would
split one response into two. Such a value is refused rather than repaired:
application/octet-stream renders nothing and injects nothing.

charsetFor and typeAndCharset disagreed for "text/html; charset=" -- one appended
nothing, the other substituted utf-8 -- so the two documentation transports
declared different encodings for one stored value, which is the divergence this
class exists to remove. Both take the same decision from the same call now,
because headerValue rebuilds the header from the parsed parts: one normalized
type, the other parameters as they were, exactly one charset. Rebuilding also
makes "; charset" and "; charset=" replaceable rather than contradictable, so
the malformed forms are no longer emitted at all.

With rebuilding, the first *usable* charset became the right one to read rather
than simply the first. While this appended, the first mattered, because that is
the one a first-wins recipient keeps; now that exactly one is emitted, serving
"charset=; charset=iso-8859-1" as utf-8 would garble a page that says plainly
what it is. My own test caught that.

Also from the review: handleClient's error path called sendError without
outputStarted, so a failure while writing the body -- a dropped connection being
the common one -- appended a second status line to a response that had already
claimed 200. It reports whether the response had started now, as the other call
sites in this file already do.

The tests move to Truth, which ARCHITECTURE.md requires and this file was not
using, and the comment volume comes down: the asymmetry was explained in four
separate KDocs and no longer exists to explain.

341 tests pass across :common and :app.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… the type

The sanitising check covered only the segment before the first ';', so
text/html; note=x<CR><LF>X-Injected: y passed it -- and the parameter loop then
wrote that CR/LF into the response header. Same response splitting the check was
added to stop, one segment further along, in the fix for it.

The whole stored value is checked now, and a refused value emits nothing but
application/octet-stream: its parameters are exactly where the control
characters would have been.

Test: 'a control character in a parameter is refused too', which fails against
the type-only check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>